{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train ML model on Cloud AI Platform\n",
    "\n",
    "This notebook shows how to:\n",
    "* Export training code from [a Keras notebook](../06_feateng_keras/solution/taxifare_fc.ipynb) into a trainer file\n",
    "* Create a Docker container based on a [DLVM container](https://cloud.google.com/ai-platform/deep-learning-containers/docs/kubernetes-container])\n",
    "* Deploy training job to cluster\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TODO: Export the data from BigQuery to GCS\n",
    "1. Navigate to [export_data.ipynb](export_data.ipynb)\n",
    "2. Update 'your-gcs-project-here' to your GCP project name\n",
    "3. Run all the notebook cells"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TODO: Edit notebook parameters\n",
    "1. Navigate to [notebook_params.yaml](notebook_params.yaml)\n",
    "2. Replace the bucket name with your own bucket containing your model (likely gcp-project with -ml at the end)\n",
    "3. Save the notebook\n",
    "4. Return to this notebook and continue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Export code from notebook\n",
    "\n",
    "This notebook extracts code from a notebook and creates a Python file suitable for use as model.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import nbformat\n",
    "import sys\n",
    "import yaml\n",
    "\n",
    "def write_parameters(cell_source, params_yaml, outfp):\n",
    "    with open(params_yaml, 'r') as ifp:\n",
    "        y = yaml.safe_load(ifp)\n",
    "        # print out all the lines in notebook\n",
    "        write_code(cell_source, 'PARAMS from notebook', outfp)\n",
    "        # print out YAML file; this will override definitions above\n",
    "        formats = [\n",
    "            '{} = {}', # for integers and floats\n",
    "            '{} = \"{}\"', # for strings\n",
    "        ]\n",
    "        write_code(\n",
    "            '\\n'.join([\n",
    "                formats[type(value) is str].format(key, value) for key, value in y.items()]),\n",
    "            'PARAMS from YAML',\n",
    "            outfp\n",
    "        )\n",
    "\n",
    "def write_code(cell_source, comment, outfp):\n",
    "    lines = cell_source.split('\\n')\n",
    "    if len(lines) > 0 and lines[0].startswith('%%'):\n",
    "        prefix = '#'\n",
    "    else:\n",
    "        prefix = ''\n",
    "    \n",
    "    print(\"### BEGIN {} ###\".format(comment), file=outfp)\n",
    "    for line in lines:\n",
    "        line = prefix + line.replace('print(', 'logging.info(')\n",
    "        if len(line) > 0 and (line[0] == '!' or line[0] == '%'):\n",
    "            print('#' + line, file=outfp)\n",
    "        else:\n",
    "            print(line, file=outfp)\n",
    "    print(\"### END {} ###\\n\".format(comment), file=outfp)\n",
    "            \n",
    "def convert_notebook(notebook_filename, params_yaml, outfp):\n",
    "    write_code('import logging', 'code added by notebook conversion', outfp)\n",
    "    with open(INPUT) as ifp:\n",
    "        nb = nbformat.reads(ifp.read(), nbformat.NO_CONVERT)\n",
    "        for cell in nb.cells:\n",
    "            if cell.cell_type == 'code':\n",
    "                if 'tags' in cell.metadata and 'display' in cell.metadata.tags:\n",
    "                    logging.info('Ignoring cell # {} with display tag'.format(cell.execution_count))\n",
    "                elif 'tags' in cell.metadata and 'parameters' in cell.metadata.tags:\n",
    "                    logging.info('Writing params cell # {}'.format(cell.execution_count))\n",
    "                    write_parameters(cell.source, PARAMS, outfp)\n",
    "                else:\n",
    "                    logging.info('Writing model cell # {}'.format(cell.execution_count))\n",
    "                    write_code(cell.source, 'Cell #{}'.format(cell.execution_count), outfp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "INPUT='../../06_feateng_keras/solution/taxifare_fc.ipynb'\n",
    "PARAMS='./notebook_params.yaml'\n",
    "OUTDIR='./container/trainer'\n",
    "\n",
    "!mkdir -p $OUTDIR\n",
    "OUTFILE=os.path.join(OUTDIR, 'model.py')\n",
    "!touch $OUTDIR/__init__.py\n",
    "with open(OUTFILE, 'w') as ofp:\n",
    "    #convert_notebook(INPUT, PARAMS, sys.stdout)\n",
    "    convert_notebook(INPUT, PARAMS, ofp)\n",
    "#!cat $OUTFILE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Try out model file\n",
    "\n",
    "<b>Note</b> Once the training starts, __Interrupt the Kernel__ (from the notebook ribbon bar above). Because it processes the entire dataset, this will take a long time on the relatively small machine on which you are running Notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019-09-23 03:30:16.322199: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  AVX2 FMA\n",
      "To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2019-09-23 03:30:16.330935: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz\n",
      "2019-09-23 03:30:16.331268: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55d16e6c9710 executing computations on platform Host. Devices:\n",
      "2019-09-23 03:30:16.331305: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>\n",
      "2019-09-23 03:30:16.331636: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n",
      "Model: \"model\"\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "dropoff_latitude (InputLayer)   [(None,)]            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "dropoff_longitude (InputLayer)  [(None,)]            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "pickup_longitude (InputLayer)   [(None,)]            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "pickup_latitude (InputLayer)    [(None,)]            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "pickup_datetime (InputLayer)    [(None,)]            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "scale_dropoff_latitude (Lambda) (None,)              0           dropoff_latitude[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "scale_dropoff_longitude (Lambda (None,)              0           dropoff_longitude[0][0]          \n",
      "__________________________________________________________________________________________________\n",
      "euclidean (Lambda)              (None,)              0           pickup_longitude[0][0]           \n",
      "                                                                 pickup_latitude[0][0]            \n",
      "                                                                 dropoff_longitude[0][0]          \n",
      "                                                                 dropoff_latitude[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "hourofday (Lambda)              (None,)              0           pickup_datetime[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "passenger_count (InputLayer)    [(None,)]            0                                            \n",
      "__________________________________________________________________________________________________\n",
      "scale_pickup_latitude (Lambda)  (None,)              0           pickup_latitude[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "scale_pickup_longitude (Lambda) (None,)              0           pickup_longitude[0][0]           \n",
      "__________________________________________________________________________________________________\n",
      "dense_features (DenseFeatures)  (None, 130)          1000000     scale_dropoff_latitude[0][0]     \n",
      "                                                                 scale_dropoff_longitude[0][0]    \n",
      "                                                                 euclidean[0][0]                  \n",
      "                                                                 hourofday[0][0]                  \n",
      "                                                                 passenger_count[0][0]            \n",
      "                                                                 scale_pickup_latitude[0][0]      \n",
      "                                                                 scale_pickup_longitude[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "h1 (Dense)                      (None, 32)           4192        dense_features[0][0]             \n",
      "__________________________________________________________________________________________________\n",
      "h2 (Dense)                      (None, 8)            264         h1[0][0]                         \n",
      "__________________________________________________________________________________________________\n",
      "fare (Dense)                    (None, 1)            9           h2[0][0]                         \n",
      "==================================================================================================\n",
      "Total params: 1,004,465\n",
      "Trainable params: 1,004,465\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n",
      "Epoch 1/5\n",
      "2019-09-23 03:30:19.192346: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1483] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.\n"
     ]
    }
   ],
   "source": [
    "!python3 $OUTFILE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Docker container\n",
    "\n",
    "Package up the trainer file into a Docker container and submit the image.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting container/Dockerfile\n"
     ]
    }
   ],
   "source": [
    "%%writefile container/Dockerfile\n",
    "FROM gcr.io/deeplearning-platform-release/tf2-cpu\n",
    "\n",
    "#RUN python3 -m pip install --upgrade --quiet tf-nightly-2.0-preview\n",
    "RUN python3 -m pip install --upgrade --quiet cloudml-hypertune\n",
    "\n",
    "COPY trainer /trainer\n",
    "CMD [\"python3\", \"/trainer/model.py\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting container/push_docker.sh\n"
     ]
    }
   ],
   "source": [
    "%%writefile container/push_docker.sh\n",
    "export PROJECT_ID=$(gcloud config list project --format \"value(core.project)\")\n",
    "export IMAGE_REPO_NAME=serverlessml_training_container\n",
    "#export IMAGE_TAG=$(date +%Y%m%d_%H%M%S)\n",
    "#export IMAGE_URI=gcr.io/$PROJECT_ID/$IMAGE_REPO_NAME:$IMAGE_TAG\n",
    "export IMAGE_URI=gcr.io/$PROJECT_ID/$IMAGE_REPO_NAME\n",
    "\n",
    "echo \"Building  $IMAGE_URI\"\n",
    "docker build -f Dockerfile -t $IMAGE_URI ./\n",
    "echo \"Pushing $IMAGE_URI\"\n",
    "docker push $IMAGE_URI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "container\n",
      "container/Dockerfile\n",
      "container/push_docker.sh\n",
      "container/trainer\n",
      "container/trainer/model.py\n",
      "container/trainer/__init__.py\n"
     ]
    }
   ],
   "source": [
    "!find container"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Note</b>: If you get a permissions error when running push_docker.sh from Notebooks, do it from CloudShell:\n",
    "* Open [CloudShell](https://console.cloud.google.com/cloudshell) on the GCP Console\n",
    "*  ```git clone https://github.com/GoogleCloudPlatform/training-data-analyst```\n",
    "*  ```cd training-data-analyst/quests/serverlessml/07_caip/solution/container```\n",
    "*  ```bash push_docker.sh```\n",
    "\n",
    "This next step takes 5 - 10 minutes to run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Building  gcr.io/qwiklabs-gcp-bdc77450c97b4bf6/serverlessml_training_container\n",
      "Sending build context to Docker daemon  17.92kB\n",
      "Step 1/4 : FROM gcr.io/deeplearning-platform-release/tf2-cpu\n",
      " ---> 5fb6bb1ebda9\n",
      "Step 2/4 : COPY trainer /trainer\n",
      " ---> 8eb46e62f49d\n",
      "Step 3/4 : RUN apt update &&     apt install --yes python3-pip &&     pip3 install --upgrade --quiet tf-nightly-2.0-preview\n",
      " ---> Running in 5d9135ef7e5a\n",
      "\u001b[91m\n",
      "WARNING: apt does not have a stable CLI interface. Use with caution in scripts.\n",
      "\n",
      "\u001b[0mGet:1 http://packages.cloud.google.com/apt cloud-sdk-bionic InRelease [6372 B]\n",
      "Get:2 http://packages.cloud.google.com/apt cloud-sdk-bionic/main amd64 Packages [86.7 kB]\n",
      "Get:3 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]\n",
      "Get:4 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]\n",
      "Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [662 kB]\n",
      "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]\n",
      "Get:7 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]\n",
      "Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [769 kB]\n",
      "Get:9 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]\n",
      "Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]\n",
      "Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [9585 B]\n",
      "Get:12 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [5230 B]\n",
      "Get:13 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]\n",
      "Get:14 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]\n",
      "Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [959 kB]\n",
      "Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [8284 B]\n",
      "Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [20.3 kB]\n",
      "Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1287 kB]\n",
      "Get:19 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [4227 B]\n",
      "Get:20 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [2496 B]\n",
      "Fetched 17.2 MB in 10s (1761 kB/s)\n",
      "Reading package lists...\n",
      "Building dependency tree...\n",
      "Reading state information...\n",
      "41 packages can be upgraded. Run 'apt list --upgradable' to see them.\n",
      "\u001b[91m\n",
      "WARNING: apt does not have a stable CLI interface. Use with caution in scripts.\n",
      "\n",
      "\u001b[0mReading package lists...\n",
      "Building dependency tree...\n",
      "Reading state information...\n",
      "The following additional packages will be installed:\n",
      "  dh-python libexpat1 libexpat1-dev libpython3-dev libpython3.6\n",
      "  libpython3.6-dev libpython3.6-minimal libpython3.6-stdlib python-pip-whl\n",
      "  python3-asn1crypto python3-cffi-backend python3-crypto python3-cryptography\n",
      "  python3-dev python3-distutils python3-idna python3-keyring\n",
      "  python3-keyrings.alt python3-lib2to3 python3-pkg-resources\n",
      "  python3-secretstorage python3-setuptools python3-six python3-wheel\n",
      "  python3-xdg python3.6 python3.6-dev python3.6-minimal\n",
      "Suggested packages:\n",
      "  python-crypto-doc python-cryptography-doc python3-cryptography-vectors\n",
      "  gnome-keyring libkf5wallet-bin gir1.2-gnomekeyring-1.0\n",
      "  python-secretstorage-doc python-setuptools-doc python3.6-venv python3.6-doc\n",
      "  binfmt-support\n",
      "The following NEW packages will be installed:\n",
      "  dh-python libexpat1-dev libpython3-dev libpython3.6-dev python-pip-whl\n",
      "  python3-asn1crypto python3-cffi-backend python3-crypto python3-cryptography\n",
      "  python3-dev python3-distutils python3-idna python3-keyring\n",
      "  python3-keyrings.alt python3-lib2to3 python3-pip python3-pkg-resources\n",
      "  python3-secretstorage python3-setuptools python3-six python3-wheel\n",
      "  python3-xdg python3.6-dev\n",
      "The following packages will be upgraded:\n",
      "  libexpat1 libpython3.6 libpython3.6-minimal libpython3.6-stdlib python3.6\n",
      "  python3.6-minimal\n",
      "6 upgraded, 23 newly installed, 0 to remove and 35 not upgraded.\n",
      "Need to get 54.2 MB of archives.\n",
      "After this operation, 89.1 MB of additional disk space will be used.\n",
      "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libexpat1 amd64 2.2.5-3ubuntu0.2 [80.5 kB]\n",
      "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython3.6 amd64 3.6.8-1~18.04.2 [1414 kB]\n",
      "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3.6 amd64 3.6.8-1~18.04.2 [202 kB]\n",
      "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython3.6-stdlib amd64 3.6.8-1~18.04.2 [1710 kB]\n",
      "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3.6-minimal amd64 3.6.8-1~18.04.2 [1609 kB]\n",
      "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython3.6-minimal amd64 3.6.8-1~18.04.2 [532 kB]\n",
      "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lib2to3 all 3.6.8-1~18.04 [76.5 kB]\n",
      "Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-distutils all 3.6.8-1~18.04 [141 kB]\n",
      "Get:9 http://archive.ubuntu.com/ubuntu bionic/main amd64 dh-python all 3.20180325ubuntu2 [89.2 kB]\n",
      "Get:10 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libexpat1-dev amd64 2.2.5-3ubuntu0.2 [122 kB]\n",
      "Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython3.6-dev amd64 3.6.8-1~18.04.2 [44.8 MB]\n",
      "Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libpython3-dev amd64 3.6.7-1~18.04 [7328 B]\n",
      "Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python-pip-whl all 9.0.1-2.3~ubuntu1.18.04.1 [1653 kB]\n",
      "Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB]\n",
      "Get:15 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB]\n",
      "Get:16 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-crypto amd64 2.6.1-8ubuntu2 [244 kB]\n",
      "Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-idna all 2.6-1 [32.5 kB]\n",
      "Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-six all 1.11.0-2 [11.4 kB]\n",
      "Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB]\n",
      "Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3.6-dev amd64 3.6.8-1~18.04.2 [508 kB]\n",
      "Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-dev amd64 3.6.7-1~18.04 [1288 B]\n",
      "Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-secretstorage all 2.3.1-2 [12.1 kB]\n",
      "Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-keyring all 10.6.0-1 [26.7 kB]\n",
      "Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-keyrings.alt all 3.0-1 [16.6 kB]\n",
      "Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-pip all 9.0.1-2.3~ubuntu1.18.04.1 [114 kB]\n",
      "Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pkg-resources all 39.0.1-2 [98.8 kB]\n",
      "Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB]\n",
      "Get:28 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-wheel all 0.30.0-0.2 [36.5 kB]\n",
      "Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-xdg all 0.25-4ubuntu1 [31.4 kB]\n",
      "\u001b[91mdebconf: delaying package configuration, since apt-utils is not installed\n",
      "\u001b[0mFetched 54.2 MB in 23s (2309 kB/s)\n",
      "(Reading database ... 84724 files and directories currently installed.)\n",
      "Preparing to unpack .../00-libexpat1_2.2.5-3ubuntu0.2_amd64.deb ...\n",
      "Unpacking libexpat1:amd64 (2.2.5-3ubuntu0.2) over (2.2.5-3ubuntu0.1) ...\n",
      "Preparing to unpack .../01-libpython3.6_3.6.8-1~18.04.2_amd64.deb ...\n",
      "Unpacking libpython3.6:amd64 (3.6.8-1~18.04.2) over (3.6.8-1~18.04.1) ...\n",
      "Preparing to unpack .../02-python3.6_3.6.8-1~18.04.2_amd64.deb ...\n",
      "Unpacking python3.6 (3.6.8-1~18.04.2) over (3.6.8-1~18.04.1) ...\n",
      "Preparing to unpack .../03-libpython3.6-stdlib_3.6.8-1~18.04.2_amd64.deb ...\n",
      "Unpacking libpython3.6-stdlib:amd64 (3.6.8-1~18.04.2) over (3.6.8-1~18.04.1) ...\n",
      "Preparing to unpack .../04-python3.6-minimal_3.6.8-1~18.04.2_amd64.deb ...\n",
      "Unpacking python3.6-minimal (3.6.8-1~18.04.2) over (3.6.8-1~18.04.1) ...\n",
      "Preparing to unpack .../05-libpython3.6-minimal_3.6.8-1~18.04.2_amd64.deb ...\n",
      "Unpacking libpython3.6-minimal:amd64 (3.6.8-1~18.04.2) over (3.6.8-1~18.04.1) ...\n",
      "Selecting previously unselected package python3-lib2to3.\n",
      "Preparing to unpack .../06-python3-lib2to3_3.6.8-1~18.04_all.deb ...\n",
      "Unpacking python3-lib2to3 (3.6.8-1~18.04) ...\n",
      "Selecting previously unselected package python3-distutils.\n",
      "Preparing to unpack .../07-python3-distutils_3.6.8-1~18.04_all.deb ...\n",
      "Unpacking python3-distutils (3.6.8-1~18.04) ...\n",
      "Selecting previously unselected package dh-python.\n",
      "Preparing to unpack .../08-dh-python_3.20180325ubuntu2_all.deb ...\n",
      "Unpacking dh-python (3.20180325ubuntu2) ...\n",
      "Selecting previously unselected package libexpat1-dev:amd64.\n",
      "Preparing to unpack .../09-libexpat1-dev_2.2.5-3ubuntu0.2_amd64.deb ...\n",
      "Unpacking libexpat1-dev:amd64 (2.2.5-3ubuntu0.2) ...\n",
      "Selecting previously unselected package libpython3.6-dev:amd64.\n",
      "Preparing to unpack .../10-libpython3.6-dev_3.6.8-1~18.04.2_amd64.deb ...\n",
      "Unpacking libpython3.6-dev:amd64 (3.6.8-1~18.04.2) ...\n",
      "Selecting previously unselected package libpython3-dev:amd64.\n",
      "Preparing to unpack .../11-libpython3-dev_3.6.7-1~18.04_amd64.deb ...\n",
      "Unpacking libpython3-dev:amd64 (3.6.7-1~18.04) ...\n",
      "Selecting previously unselected package python-pip-whl.\n",
      "Preparing to unpack .../12-python-pip-whl_9.0.1-2.3~ubuntu1.18.04.1_all.deb ...\n",
      "Unpacking python-pip-whl (9.0.1-2.3~ubuntu1.18.04.1) ...\n",
      "Selecting previously unselected package python3-asn1crypto.\n",
      "Preparing to unpack .../13-python3-asn1crypto_0.24.0-1_all.deb ...\n",
      "Unpacking python3-asn1crypto (0.24.0-1) ...\n",
      "Selecting previously unselected package python3-cffi-backend.\n",
      "Preparing to unpack .../14-python3-cffi-backend_1.11.5-1_amd64.deb ...\n",
      "Unpacking python3-cffi-backend (1.11.5-1) ...\n",
      "Selecting previously unselected package python3-crypto.\n",
      "Preparing to unpack .../15-python3-crypto_2.6.1-8ubuntu2_amd64.deb ...\n",
      "Unpacking python3-crypto (2.6.1-8ubuntu2) ...\n",
      "Selecting previously unselected package python3-idna.\n",
      "Preparing to unpack .../16-python3-idna_2.6-1_all.deb ...\n",
      "Unpacking python3-idna (2.6-1) ...\n",
      "Selecting previously unselected package python3-six.\n",
      "Preparing to unpack .../17-python3-six_1.11.0-2_all.deb ...\n",
      "Unpacking python3-six (1.11.0-2) ...\n",
      "Selecting previously unselected package python3-cryptography.\n",
      "Preparing to unpack .../18-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ...\n",
      "Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ...\n",
      "Selecting previously unselected package python3.6-dev.\n",
      "Preparing to unpack .../19-python3.6-dev_3.6.8-1~18.04.2_amd64.deb ...\n",
      "Unpacking python3.6-dev (3.6.8-1~18.04.2) ...\n",
      "Selecting previously unselected package python3-dev.\n",
      "Preparing to unpack .../20-python3-dev_3.6.7-1~18.04_amd64.deb ...\n",
      "Unpacking python3-dev (3.6.7-1~18.04) ...\n",
      "Selecting previously unselected package python3-secretstorage.\n",
      "Preparing to unpack .../21-python3-secretstorage_2.3.1-2_all.deb ...\n",
      "Unpacking python3-secretstorage (2.3.1-2) ...\n",
      "Selecting previously unselected package python3-keyring.\n",
      "Preparing to unpack .../22-python3-keyring_10.6.0-1_all.deb ...\n",
      "Unpacking python3-keyring (10.6.0-1) ...\n",
      "Selecting previously unselected package python3-keyrings.alt.\n",
      "Preparing to unpack .../23-python3-keyrings.alt_3.0-1_all.deb ...\n",
      "Unpacking python3-keyrings.alt (3.0-1) ...\n",
      "Selecting previously unselected package python3-pip.\n",
      "Preparing to unpack .../24-python3-pip_9.0.1-2.3~ubuntu1.18.04.1_all.deb ...\n",
      "Unpacking python3-pip (9.0.1-2.3~ubuntu1.18.04.1) ...\n",
      "Selecting previously unselected package python3-pkg-resources.\n",
      "Preparing to unpack .../25-python3-pkg-resources_39.0.1-2_all.deb ...\n",
      "Unpacking python3-pkg-resources (39.0.1-2) ...\n",
      "Selecting previously unselected package python3-setuptools.\n",
      "Preparing to unpack .../26-python3-setuptools_39.0.1-2_all.deb ...\n",
      "Unpacking python3-setuptools (39.0.1-2) ...\n",
      "Selecting previously unselected package python3-wheel.\n",
      "Preparing to unpack .../27-python3-wheel_0.30.0-0.2_all.deb ...\n",
      "Unpacking python3-wheel (0.30.0-0.2) ...\n",
      "Selecting previously unselected package python3-xdg.\n",
      "Preparing to unpack .../28-python3-xdg_0.25-4ubuntu1_all.deb ...\n",
      "Unpacking python3-xdg (0.25-4ubuntu1) ...\n",
      "Setting up python-pip-whl (9.0.1-2.3~ubuntu1.18.04.1) ...\n",
      "Setting up libexpat1:amd64 (2.2.5-3ubuntu0.2) ...\n",
      "Processing triggers for mime-support (3.60ubuntu1) ...\n",
      "Setting up python3-cffi-backend (1.11.5-1) ...\n",
      "Setting up python3-crypto (2.6.1-8ubuntu2) ...\n",
      "Setting up python3-idna (2.6-1) ...\n",
      "Setting up python3-xdg (0.25-4ubuntu1) ...\n",
      "Setting up python3-six (1.11.0-2) ...\n",
      "Setting up python3-wheel (0.30.0-0.2) ...\n",
      "Setting up python3-pkg-resources (39.0.1-2) ...\n",
      "Setting up libpython3.6-minimal:amd64 (3.6.8-1~18.04.2) ...\n",
      "Setting up python3-asn1crypto (0.24.0-1) ...\n",
      "Processing triggers for libc-bin (2.27-3ubuntu1) ...\n",
      "Setting up libexpat1-dev:amd64 (2.2.5-3ubuntu0.2) ...\n",
      "Setting up python3-lib2to3 (3.6.8-1~18.04) ...\n",
      "Setting up python3-distutils (3.6.8-1~18.04) ...\n",
      "Setting up python3-cryptography (2.1.4-1ubuntu1.3) ...\n",
      "Setting up libpython3.6-stdlib:amd64 (3.6.8-1~18.04.2) ...\n",
      "Setting up python3-keyrings.alt (3.0-1) ...\n",
      "Setting up python3.6-minimal (3.6.8-1~18.04.2) ...\n",
      "Setting up python3-pip (9.0.1-2.3~ubuntu1.18.04.1) ...\n",
      "Setting up python3-setuptools (39.0.1-2) ...\n",
      "Setting up python3-secretstorage (2.3.1-2) ...\n",
      "Setting up dh-python (3.20180325ubuntu2) ...\n",
      "Setting up libpython3.6:amd64 (3.6.8-1~18.04.2) ...\n",
      "Setting up python3.6 (3.6.8-1~18.04.2) ...\n",
      "Setting up python3-keyring (10.6.0-1) ...\n",
      "Setting up libpython3.6-dev:amd64 (3.6.8-1~18.04.2) ...\n",
      "Setting up python3.6-dev (3.6.8-1~18.04.2) ...\n",
      "Setting up libpython3-dev:amd64 (3.6.7-1~18.04) ...\n",
      "Setting up python3-dev (3.6.7-1~18.04) ...\n",
      "Processing triggers for libc-bin (2.27-3ubuntu1) ...\n",
      "Removing intermediate container 5d9135ef7e5a\n",
      " ---> c25211bc8d1f\n",
      "Step 4/4 : CMD [\"python3\", \"/trainer/model.py\"]\n",
      " ---> Running in 4d1cddc80dce\n",
      "Removing intermediate container 4d1cddc80dce\n",
      " ---> 005b3a997797\n",
      "Successfully built 005b3a997797\n",
      "Successfully tagged gcr.io/qwiklabs-gcp-bdc77450c97b4bf6/serverlessml_training_container:latest\n",
      "Pushing gcr.io/qwiklabs-gcp-bdc77450c97b4bf6/serverlessml_training_container\n",
      "The push refers to repository [gcr.io/qwiklabs-gcp-bdc77450c97b4bf6/serverlessml_training_container]\n",
      "9781db4bf0e2: Preparing\n",
      "8150a3c43d14: Preparing\n",
      "a71e72a550ba: Preparing\n",
      "c23179f29c5e: Preparing\n",
      "7e4890a23200: Preparing\n",
      "54d2eb300334: Preparing\n",
      "9464e4075ebf: Preparing\n",
      "d811f4e3bddb: Preparing\n",
      "219acea3b74e: Preparing\n",
      "add8d5b9d28b: Preparing\n",
      "aa35d6dde7c2: Preparing\n",
      "d6a152cedf91: Preparing\n",
      "920c1e5a5ee9: Preparing\n",
      "0cfda733c555: Preparing\n",
      "a0e77507151e: Preparing\n",
      "baf51ba9f2bc: Preparing\n",
      "122be11ab4a2: Preparing\n",
      "7beb13bce073: Preparing\n",
      "f7eae43028b3: Preparing\n",
      "6cebf3abed5f: Preparing\n",
      "d6a152cedf91: Waiting\n",
      "920c1e5a5ee9: Waiting\n",
      "0cfda733c555: Waiting\n",
      "a0e77507151e: Waiting\n",
      "baf51ba9f2bc: Waiting\n",
      "122be11ab4a2: Waiting\n",
      "7beb13bce073: Waiting\n",
      "f7eae43028b3: Waiting\n",
      "6cebf3abed5f: Waiting\n",
      "54d2eb300334: Waiting\n",
      "9464e4075ebf: Waiting\n",
      "d811f4e3bddb: Waiting\n",
      "219acea3b74e: Waiting\n",
      "aa35d6dde7c2: Waiting\n",
      "add8d5b9d28b: Waiting\n",
      "c23179f29c5e: Layer already exists\n",
      "7e4890a23200: Layer already exists\n",
      "a71e72a550ba: Layer already exists\n",
      "54d2eb300334: Layer already exists\n",
      "d811f4e3bddb: Layer already exists\n",
      "9464e4075ebf: Layer already exists\n",
      "219acea3b74e: Layer already exists\n",
      "add8d5b9d28b: Layer already exists\n",
      "aa35d6dde7c2: Layer already exists\n",
      "d6a152cedf91: Layer already exists\n",
      "920c1e5a5ee9: Layer already exists\n",
      "0cfda733c555: Layer already exists\n",
      "a0e77507151e: Layer already exists\n",
      "122be11ab4a2: Layer already exists\n",
      "baf51ba9f2bc: Layer already exists\n",
      "6cebf3abed5f: Layer already exists\n",
      "7beb13bce073: Layer already exists\n",
      "f7eae43028b3: Layer already exists\n",
      "8150a3c43d14: Pushed\n",
      "9781db4bf0e2: Pushed\n",
      "latest: digest: sha256:46eb0ddf5851cfb187f87c974b31dff77b89bb3111550235fa92c1cc8863cf74 size: 4505\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "cd container\n",
    "bash push_docker.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deploy to AI Platform\n",
    "\n",
    "Submit a training job using this custom container that we have just built. After you submit the job, [monitor it here](https://console.cloud.google.com/ai-platform/jobs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "jobId: serverlessml_20190923_034220\n",
      "state: QUEUED\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Job [serverlessml_20190923_034220] submitted successfully.\n",
      "Your job is still active. You may view the status of your job with the command\n",
      "\n",
      "  $ gcloud ai-platform jobs describe serverlessml_20190923_034220\n",
      "\n",
      "or continue streaming the logs with the command\n",
      "\n",
      "  $ gcloud ai-platform jobs stream-logs serverlessml_20190923_034220\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "JOBID=serverlessml_$(date +%Y%m%d_%H%M%S)\n",
    "REGION=us-central1\n",
    "PROJECT_ID=$(gcloud config list project --format \"value(core.project)\")\n",
    "BUCKET=$(gcloud config list project --format \"value(core.project)\")-ml\n",
    "\n",
    "#IMAGE=gcr.io/deeplearning-platform-release/tf2-cpu\n",
    "IMAGE=gcr.io/$PROJECT_ID/serverlessml_training_container\n",
    "\n",
    "gcloud beta ai-platform jobs submit training $JOBID \\\n",
    "   --staging-bucket=gs://$BUCKET  --region=$REGION \\\n",
    "   --master-image-uri=$IMAGE \\\n",
    "   --master-machine-type=n1-standard-4 --scale-tier=CUSTOM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The training job will take 35 - 45 minutes to complete on the dataset. You can cancel the job once you confirm it started and have inspected the logs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2020 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
